United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



I0/500.H26 



07/07/2004 



David Attwater 



23117 7590 09/01/2010 

NIXON & VANDERHYE, PC 

901 NORTH GLEBE ROAD, 1 1TH FLOOR 

ARLINGTON, VA 22203 



SAINT CYR, LEONARD 



PAPER NUMBER 



DELIVERY MODE 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/500,826 


Applicant(s) 

ATTWATER ET AL. 


Examiner 

LEONARD SAINT CYR 


Art Unit 

2626 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 06/01/10 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) |EI Claim(s) 1-5, 7- 11, 13-27, 29 -33 and 35-46 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) 13 Claim(s) 1-5. 7- 11. 13-27. 29 -33 and 35-46 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 07 July 2004 is/are: a)^ accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) ^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)E| All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) ^| Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 

Paper No(s)/Mail Date . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 20100825 



Application/Control Number: 10/500,826 
Art Unit: 2626 



Page 2 



DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments filed 06/01/10 have been fully considered but they are not 
persuasive. 

Applicant argues that neither Coffman et al., nor Gergic et al., teach storing input 
and output data indicative of one or more properties of the input and output ports and/or 
the input responses and output prompts communicated therethrough (Amendment, 
pages 18-24). 

The examiner disagrees, since Coffman et al., disclose "The DMAF comprises a 
mechanism for conveying application properties to the CVM through the DMA... Such 
properties include the resources the application needs such as engine resources 
(speech recognition, NLU, etc.) data files (such as NLU and grammar objects), the 
algorithm string for input processing (i.e., the set and order of engines needed for 
processing the use input). For example, it the user input comprises spoken utterances 
(voice command), the algorithm string may comprise: front end+speech 
recognition+NLU. If the user input is a typed command, the algorithm string may 
be just NLU, etc." (speech recognition, and NLU are considered as stored input and 
output data types; paragraph 60). 

Applicant argues that neither Coffman et al., nor Gergic et al., teach that said 
input and output type data is updated when any of said one or more properties change; 
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and/or output prompts are sent and/or input responses are received (Amendment, 
pages 18-24). 

The examiner disagrees, since Coffman et al., disclose "Another mechanism is 
provided to notify the DMA (and possibly other CVM components) when one or more of 
the application properties change. For example, a Task manager (which is a CVM 
component) should be notified of changes to the application properties. As described 
below, the Task manager is a CVM component that communicates with the 
conversational engines and, thus, needs to know the algorithm string of the user 
input and when such string is modified so that the Task manager can instantiate 
and use the proper engines for processing such user input" (speech recognition, 
and NLU are considered as the updated input and output type data; paragraphs 60, and 
61). 

2. Applicant's arguments with respect to claims 1 -5, 7 - 1 1 , 1 3 - 27, 29 - 33, 35 - 
46 have been considered but are moot in view of the new ground(s) of rejection. 

Applicant argues that neither Coffman et al., nor Gergic et al., teach providing an 
interactive dialogue apparatus for selecting the most appropriate mode for input from or 
output to the user, depending on the assessment of the user's preferences 
(Amendment, pages 18 - 24). 



Claim Rejections - 35 USC § 101 
3. 35 U.S.C. 101 reads as follows: 
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Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 45, and 46 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. Claims 45, and 46 are directed to 
a computer readable medium storing processor executable instructions that is not 
limited to a non transitory, and thus, statutory medium. The scope of "computer- 
readable medium" since not defined in the specification may include signal-based 
mediums such as "signals used to propagate instructions", "carrier waves/pulses", etc . 
A signal does not fall within one of the four statutory categories of invention (i.e., 
process, machine, manufacture, or composition of matter) because it is an ephemeral, 
transient signal and thus is non-statutory. Since the scope of "computer-readable 
medium" may include these non-statutory instances, claims 45, and 46 are directed to 
non-statutory subject matter. 

Claim Rejections - 35 USC § 103 

3. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

4. Claims 1 -5, 7 - 1 1 , 13 - 27, 29 - 33, 35-46 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Coffman et al., (US PAP 2003/0005174) in view of 
Lucassen et al., (US PAP 2003/0023953). 

As per claims 1 , and 23, Coffman et al., teach an interactive dialogue that 
comprises: 
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at least one input port; two or more output ports ("converts the abstract output 
event into one or more modalities for presentation to the user"; paragraph 22, lines 6 - 
8; paragraph 60, lines 9-12); 

means for processing input responses to determine the semantic meaning 
thereof (paragraph 94, line 5); 

and control means for determining a suitable output prompt to be output from at 
least one of said output ports in response to a received input response ("sending output 
events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153); 

wherein said output ports are respectively arranged to output prompts of different 
types ("pen recognition, speech recognition, TTS"; paragraph 153); 

a first store storing input and output type data indicative of one or more 
properties of the input and output ports and/or the input responses and output prompts 
communicated there through ("The DMA will also manage the output of these 
methods by passing it to the appropriate components for processing, using an 
algorithm string, similar to the algorithm string used for input handling (as 
described below), to control the response processing and generation by the 
appropriate engines... the algorithm string may comprise: front end+speech 
recognition+NLU. If the user input is a typed command, the algorithm string may 
be just NLU, etc"; paragraphs 53, 60, and 61); 

wherein said input and output type data is updated when: i) any of said one or 
more properties change; and/or ii) output prompts are sent; and/or iii) input responses 
are received ("The DMA will also manage the output of these methods by passing 
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it to the appropriate components for processing, using an algorithm string, 
similar to the algorithm string used for input handling (as described below), to 
control the response processing and generation by the appropriate engines. ..if 

the user input comprises spoken utterances (voice command), the algorithm string 
may comprise: front end+speech recognition+NLU. If the user input is a typed 
command, the algorithm string may be just NLU, etc "; paragraphs 53, 60, and 61). 

However, Coffman et al., do not specifically teach wherein one of said properties 
is the utilization made by a user of each input and output port; and means for 
establishing from said properties for each of said input and output ports a user 
preference value. 

Lucassen et al., teach that key aspects of this form of interaction include the 
ability of the system to use the best possible combination of interface modalities based 
on the user's current preference needs and abilities as well as the application 
requirements and device capabilities. At the same time, the system is characterized by 
the ability to dynamically update its choice of modalities based on what the user 
chooses to do. Thus, upon failure of the user to respond to a spoken prompt, the 
system might choose to revert to a visual interface-an implicit assumption that the user 
is in environment where speech interaction is inappropriate-equivalently, a spoken 
request from the user might cause the system to update its behavior to switch from 
visual to spoken interaction (paragraph 41). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to dynamically update modalities base on user's current 
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preference as taught by Lucassen et al., in Coffman et al., because that would make the 
system to update its behavior to switch from visual to spoken interaction (paragraph 41 , 
last two lines) 

As per claims 2, and 24, Coffman et al., teach an interactive dialogue that 
comprises: 

two or more input ports; at least one output port ("voice command, and typed 
command"; paragraph 60; paragraph 22); 

means for processing input responses received at one or more of said input ports 
to determine the semantic meaning thereof(paragraph 94, line 5); 

and control means for determining a suitable output prompt to be output from at 
least one of said output ports in response to a received input response ("sending output 
events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153); 

wherein said input ports are respectively arranged to receive input responses of 
different types ("voice command, and typed command"; paragraph 60); the apparatus 
and method further comprising 

a first store storing input and output type data indicative of one or more 
properties of the input and output ports and/or the input responses and output prompts 
communicated there through ("The DMA will also manage the output of these 
methods by passing it to the appropriate components for processing, using an 
algorithm string, similar to the algorithm string used for input handling (as 
described below), to control the response processing and generation by the 
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appropriate engines... the algorithm string may comprise: front end+speech 
recognition+NLU. If the user input is a typed command, the algorithm string may 
be just NLU, etc"; paragraphs 53, and 60); 

wherein said input and output type data is updated when: i) any of said one or 
more properties change; and/or ii) output prompts are sent; and/or iii) input responses 
are received ( "The DMA will also manage the output of these methods by passing 
it to the appropriate components for processing, using an algorithm string, 
similar to the algorithm string used for input handling (as described below), to 
control the response processing and generation by the appropriate engines. ..if 
the user input comprises spoken utterances (voice command), the algorithm string 
may comprise: front end+speech recognition+NLU. If the user input is a typed 
command, the algorithm string may be just NLU, etc"; paragraphs 53, and 60). 

However, Coffman et al., do not specifically teach wherein one of said properties 
is the utilization made by a user of each input and output port; and means for 
establishing from said properties for each of said input and output ports a user 
preference value. 

Lucassen et al., teach that key aspects of this form of interaction include the 
ability of the system to use the best possible combination of interface modalities based 
on the user's current preference needs and abilities as well as the application 
requirements and device capabilities. At the same time, the system is characterized by 
the ability to dynamically update its choice of modalities based on what the user 
chooses to do. Thus, upon failure of the user to respond to a spoken prompt, the 
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system might choose to revert to a visual interface-an implicit assumption that the user 
is in environment where speech interaction is inappropriate-equivalently, a spoken 
request from the user might cause the system to update its behavior to switch from 
visual to spoken interaction (paragraph 41). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to dynamically update modalities base on user's current 
preference as taught by Lucassen et al., in Coffman et al., because that would make the 
system to update its behavior to switch from visual to spoken interaction (paragraph 41 , 
last two lines) 

As per claims 3 and 25, Coffman et al., in view of Lucassen et al., further 
disclose at least one additional output port, wherein said control means is further 
arranged to determine a suitable output prompt to be output from at least one of said 
output ports in response to a received input response ("sending output events to the 
appropriate engine"); and wherein said output ports are respectively arranged to output 
prompts of different types (Coffman et al., "pen recognition, speech recognition, TTS"; 
paragraph 102, lines 4-7; paragraph 153). 

As per claims 4 and 26, Coffman et al., in view of Lucassen et al., further 
disclose that for any particular received input prompt, output prompts which are 
semantically synonymous ("semantic meaning") or which mutually contribute towards a 
single semantic message independent of type are output from two or more of the output 
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ports (Coffman et al., "converts the abstract output event into one or more modalities for 
presentation to the user"; paragraph 94, line 5; paragraph 22, lines 6 - 8). 

As per claims 5 and 27, Coffman et al., in view of Lucassen et al., further 
disclose that each input or output port is adapted to connect to one or more input or 
output devices via respective device gateways (Coffman et al., "input and output 
devices"; paragraph 172; paragraph 101, line 2). 

As per claims 7, and 29, Coffman et al., in view of Lucassen et al., further 
disclose that one of said properties is the connection of appropriate input or output 
devices to each of said input or output ports (Coffman et al., "properties includes the 
resources the application needs for processing the user input"; paragraph 60). 

As per claims 8, and 30, Coffman et al., in view of Lucassen et al., further 
disclose that one of said properties is user preference value for each of said input and 
output ports ("key aspects of this form of interaction include the ability of the system to 
use the best possible combination of interface modalities based on the user's current 
preference needs and abilities as well as the application requirements and device 
capabilities; paragraph 41) 

As per claims 9, and 31 , Coffman et al., in view of Lucassen et al., further 
disclose that one of said properties is device property data of input or output devices 
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connected to said input or output ports (Coffman et al., "voice command and typed 
command"; paragraph 60, lines 9- 12). 

As per claims 10 and 32, Coffman et al., in view of Lucassen et al., further 
disclose that one of said properties is implementation data indicative of: whether an 
output prompt has been implemented in each output prompt type and/or input parse 
rules for each input response type (Coffman et al., "NLU parse tree... and data 
associated with a NLU process"; paragraph 103, lines 1 -4; paragraph 122). 

As per claims 1 1 and 33, Coffman et al., in view of Lucassen et al., further 
disclose that one of said properties is type-supported data indicative of whether the 
apparatus is capable of receiving and/or outputting input responses and/or output 
prompts of each type (Coffman et al., "converts the abstract output event into one or 
more modalities for presentation to the user"; paragraph 22). 

As per claims 13, and 35, Coffman et al., in view of Lucassen et al., further 
disclose that the update of said data comprises instantiating new data structures 
("hierarchical tree structure") to store the values of said changed properties, and storing 
said previous data to give a historical record of said data (Coffman et al., "transaction 
history"; paragraph 13, line 7; paragraph 64, lines 1 -4). 
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As per claims 14, and 36, Coffman et al., in view of Lucassen et al., further 
disclose that said input and output type data further includes timing data indicative of 
the timings of changes in said one or more properties (Coffman et al., "time stamped"; 
paragraph 163). 

As per claims 15, and 37, Coffman et al., in view of Lucassen et al., further 
disclose that said input and output type data comprises a single data entry for each 
input and output type, the value taken by a particular data entry being dependent on 
previous values of any one or more of that or other data entries (Coffman et al., "top 
scoring query result"; paragraphs 66, and 122). 

As per claims 16, and 38, Coffman et al., in view of Lucassen et al., further 
disclose a second store data defining a dialogue to be held with a user, and dialogue 
progression conditions which must be met to allow a user to progress through the 
dialogue, at least some of said conditions involving the stored input and output type 
data (Coffman et al., "for any given user input, arbitration mechanism will determine the 
target DMA instance managing the associated sub-dialog"; paragraphs 48, 49, and 59). 

As per claims 17, and 39, Coffman et al., in view of Lucassen et al., further 
disclose a second store storing data defining a dialogue model comprising an initial 
state, a plurality of subsequent states, possible transitions between said states, and for 
each transition at least one associated condition to be satisfied before that transition is 
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deemed allowable, at least some of said conditions involving the stored input and output 
type data (Coffman et al., "hierarchical tree structure that contains root, parent and 
children nodes"; paragraph 13; paragraph 49). 

As per claims 18, and 40, Coffman et al., in view of Lucassen et al., further 
disclose that the second store comprises a plurality of distributed storage media 
(Coffman et al., "multimedia streams"; paragraph 104, line 15) 

As per claims 19, and 41 , Coffman et al., in view of Lucassen et al., further 
disclose port control means for controlling the connections of input or output devices to 
said input or output ports in response to the stored input and output type data (Coffman 
et al., "a mechanism for sending output events to the appropriate engine"; paragraph 
153, lines 4 -6). 

As per claims 20, and 42, Coffman et al., in view of Lucassen et al., further 
disclose means for generating output prompts ("compose prompts"), said means being 
operable to generate output prompts adapted for particular output ports in dependence 
on the stored input and output type data (Coffman et al., "a mechanism for sending 
output events to the appropriate engine"; paragraph 102, lines 4-7; paragraph 153, 
lines 4 -6). 
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As per claims 21 , and 43, Coffman et al., in view of Lucassen et al., further 
disclose that first store comprises a plurality of distributed storage media each logically 
interconnected (Coffman et al., "multimedia streams"; paragraph 104, line 15). 

As per claims 22, and 44, Coffman et al., in view of Lucassen et al., further 
disclose that the different types of output prompts or input responses comprise audio 
prompts or responses, or visual prompts or responses, or motor prompts or responses, 
in any combination thereof (Coffman et al., "text-to-display or prompt is provided"; 
paragraph 166, lines 17, and 18). 

As per claim 45, and 46, Coffman et al., in view of Lucassen et al., further 
disclose that a computer program or suite of programs so arranged such that when 
loaded into a computer it or they renders the computer an apparatus according to 
claims 1, and 23 (Coffman et al., paragraph 180). 

Conclusion 

5. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to LEONARD SAINT CYR whose telephone number is 
(571) 272-4247. The examiner can normally be reached on Mon- Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on (571) 272-7602. The fax phone 
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number for the organization where this application or proceeding is assigned is (571)- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or (571) 272-1000. 
LS 

/Leonard Saint-Cyr/ 
Examiner, Art Unit 2626 



